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(54) METHOD AND DEVICE FOR DISPLAYING AND EDITING MULTIMEDIA 
INFORMATION 

(57)Abstract: 

PURPOSE: To provide a system which edits a multimedia project on a display 
picture. 

CONSTITUTION: The computer display picture presented to a user provides 
three different views of multimedia information. The first view is a player 1 1 
similar to a virtual video recorder accompanied with a monitor. The second view 
is a story board 13 which resembling a conventional story board very much. The 



third view is a strip viewer 15 which presents a view based on the time base of 
multimedia information to the user. There is an epoch-making screen tool called 
a 'splinter' on the strip viewer display part, and it is used to edit the start point and 
the end point of a displayed multimedia track. 
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CLAIMS 



[Claim{s)] 

[Claim 1] In the approach of displaying and editing multimedia information in a 
display screen The process which displays the 1st unit of multimedia information 



on said display screen; Said display screen is adjoined at the 1st unit of said 
multimedia information. The process, and the; 1st part which display the 2nd unit 
of multimedia information, SUPURINDA which consists of the 2nd part combined 
with said 1st part, and the 3rd part combined with said 1st part The 1st unit of 
said multimedia information, Said 1st part of the process which is located 
between the 2nd unit of said multimedia information, and is displayed on said 
display screen, and; SUPURINDA, The process to which said the 2nd part and 
said 3rd part are moved along with the 1st unit and 2nd unit of said multimedia 
information; It consists of the process in which the 1st unit and 2nd unit of said 
multimedia information on said display screen are edited according to a motion of 
SUPURINDA. The approach into which multimedia information is edited by 
motion of SUPURINDA on the display screen. 

[Claim 2] It combines with said digital computer, the equipment into which 
multimedia information is displayed and edited setting a digital computer 
and; -- The indicating equipment which has cursor; It combines with said digital 
computer. The 1st unit of the multimedia information displayed on the cursor 
controller which controls the location of said cursor in said display, and the; 
aforementioned display; to said display The 2nd unit, and the; 1st part of 
multimedia information which are adjoined and displayed on the 1st unit of said 
multimedia information, The cursor as which it consists of the 2nd part combined 
with said 1st part, and the 3rd part combined with said 1st part, and said 1st part, 
said 2nd part, and said 3rd part are displayed on said display screen, The 1st 
unit of said multimedia information [ in / it can move freely on the display screen 
with the cursor controller combined with said display screen and / said indicating 
equipment ], SUPURINDA displayed between the 2nd unit of said multimedia 
information; The 1st unit of the multimedia information as which said 
SUPURINDA was displayed by said cursor controller, Equipment possessing an 
edit means to answer being moved along with the 2nd unit of multimedia 
information, and to edit the 1st unit and 2nd unit of said multimedia information. 
[Claim 3] It combines with said digital computer the equipment into which 



multimedia information is displayed and edited setting a digital computer 
and; -- The indicating equipment which has cursor; It combines with said digital 
computer. The cursor controller which controls the location of said cursor in said 
display; The truck showing the multimedia information as which it is on said 
display and each is displayed by the time amount line method of at least one 
multimedia information, Each is connected with one multimedia code track. With 
cursor and a cursor controller A motion of the display which consists of each 
element of the scale which displays at least one SUPURINDA which can move 
freely, and the time base for every; multimedia code track along a multimedia 
code track, and; SUPURINDA is followed. Edit of a multimedia code track 
Equipment possessing an edit means to perform. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] Especially this invention relates to the approach and 
equipment which provide a user with the intuitive approach for editing the 
multimedia information which displays multimedia information on a display 
screen, and a display screen is shown about the field of a multimedia display and 
edit equipment. 
[0002] 

[Description of the Prior Art] Multimedia combines with a single work piece the 
media from which two or more molds differ For example, the multimedia work 
piece could consist of video information, audio information, graphic form overlay, 
and combination of animation. The multimedia work piece was developed and 
the opportunity for a computer to be used in order to display has increased all the 
more because the computer is suitable for combining such various media molds. 



Since creation of a multimedia production is relatively [ manufacture / video ] 
similar, on the occasion of creation of a multimedia production, the technique of 
video is used in many cases. For example, standard video edit tools, such as a 
storyboard and a videocassette recorder for edit, are used in many cases as a 
multimedia development tool for assisting a with creation of a multimedia 
production. However, if a computer is used, the new interactive mode tool with 
which creation and edit of a multimedia production are assisted can be created. 
Probably, as for those new interactive mode tools, it will be ideal that both 
learning and using it constitute an easy intuitive user interface. 
[0003] 

[Problem(s) to be Solved by the Invention] Therefore, the purpose of this 
invention is offering the approach and equipment which display multimedia 
information on a computer display screen. Another purpose of this invention is 
providing a user with the approach and equipment into which it enables it to edit 
the multimedia information displayed on the display screen by the intuitive 
method. Still more nearly another purpose of this invention is showing a user the 
intuitive approach which extends and trims the truck of multimedia information by 
displaying multimedia information as a strip with the tool for edit used in order to 
extend or trim a strip. 
[0004] 

[Means for Solving the Problem] The above-mentioned purpose and the other 
purposes are attained by the original approach and the equipment of this 
invention. The approach and equipment of this invention consist of systems into 
which a multimedia project is edited in a display screen. The computer display 
screen which a user is shown forms three different views of multimedia 
information. The 1st view is a player similar to the virtual videocassette recorder 
accompanied by a monitor. The 2nd view is a storyboard which is very well alike 
in the conventional storyboard. The 3rd view is a strip viewer which shows a user 
the view based on the time amount line of multimedia information. A strip viewer 
display is a screen tool called "SUPURINDA (splinder)." SUPURINDA is used in 



order to edit the start point and the ending point of a multimedia truck. The 
purpose, the description, and advantage of this invention will become clear to this 
contractor from the following detailed explanation. 
[0005] 

[Example] The approach and equipment which display multimedia information 
and are edited are indicated. During the following explanation, in order to make 
this invention understand completely, the specific vocabulary of explanation is 
mentioned for convenience, but probably, it will be clear to this contractor that 
such a specific detailed matter is not required, in order to carry out this invention. 
Moreover, in order not to make this invention unclear unnecessarily depending 
on the case, the gestalt of a block diagram may show a well-known circuit and 
equipment. 

[0006] Reference of drawing 1 shows the multimedia editor display screen using 
instruction of this invention. A multimedia editor display screen shows a user 
three the "views" of multimedia information. A player 1 1 offers the 1st view of 
multimedia information. The player 1 1 is designed so that actuation of the video 
tape recorder for edit may be simulated, therefore it is used in order to observe 
and record a live video. A user "records" video by using a player 1 1 and 
identifying, an initiation video frame (inch-point), i.e., the "Inn point", and a 
termination video frame (out-point), i.e., the "out point." The segment which 
consists of the continuous video frame which made such and was specified is 
called the "shot" of video. While the video player is not operating, the frame of 
the beginning of current video selection is displayed on an observation field. 
[0007] The storyboard 13 of drawing 1 offers the 2nd view of multimedia 
information. A storyboard 13 gives a user the view of the multimedia information 
on "higher level" by displaying all the shots of video as a single icon so that a 
shot 5 may show from a shot 1 . A user can record, delete and copy all the shots 
of video by using a storyboard 13, and choosing the icon which expresses a shot 
with cursor, next choosing a desired editing task from a menu. 
[0008] Or a storyboard 13 can be used in order to edit high-level video structures. 



such as a "scene" or an "act." These vocabulary is borrowed from the theatrical 
world, and it is used in order to define the video of higher level, or a group 
division of a film. For example, the producer/artist who creates a multimedia 
production may desire to collect a series of video shots into one a "scene." If one 
example is given, a series of shots which photoed the man and woman who are 
talking in French cafe by variously different camera angle will be able to be 
summarized as a scene of "French cafe." Then, the aggregate of a scene can be 
packed into the "act" of higher level. If it continues about a previous example, a 
series of scenes including the scene where man and woman meet several times 
first will be able to be summarized as an act of "encounter of man and woman." 
One example of the layered structure of a shot is indicated to be the act 
considered that it can use drawing 2 in order that a producer may compose a 
multimedia production, and a scene. 

[0009] However, the storyboard 13 of drawing 1 does not impose the strict 
layered structure which a producer has to use. Therefore, this storyboard 13 can 
be used for editing on the hierarchy level of arbitration which a user desires. For 
example, in order for a producer to complete the mechanics of one specific 
scene, a storyboard 13 may be used on "shot" level. It can raise to "scene" level 
and the overall feel of the mood of the multimedia production can be held. 
[0010] If two views of these starts, i.e., a player 1 1 and two views of a storyboard 
13, are combined, the functionality of the video tape recorder for edit and an 
ordinary storyboard will be acquired. Thus, the tool of the conventional video edit 
studio is given to a user. Therefore, if it is a user expert in the conventional video 
edit tool, it is possible to learn the usage of the multimedia editor explained 
above to the inside of a short time, and to use it. 

[001 1] The 3rd view of the multimedia information which the multimedia editor 
display screen of drawing 1 presents is the strip viewer 15. A detail looks at the 
contents of the media truck, so that this view was impossible at in advance. If the 
strip viewer 15 is used, a user is a time amount line method and can observe 
multimedia information with the level of the resolving power which can moreover 



be adjusted. The strip viewer 15 expresses multimedia information as the most 
detailed level (namely, the highest resolving power) as "atomic" piece which 
cannot be decomposed. For example, when a user observes video information 
on the most detailed level (namely, the highest resolving power) with the strip 
viewer 15, the frame of the video according to all individuals is displayed. 
[0012] If an example is given, the case where the strip viewer 15 of drawing 1 is 
set as the highest resolution so that each video frame may be displayed, 
respectively will be considered. The display is similar to the strip of the motion- 
picture film with which each frame has the original frame number. As for the time 
scale when displaying multimedia information, i.e., "resolving-power" scale, 
multimedia information is shown caudad. since it is the highest resolution in 
drawing 1 resolution — the scale is enumerating continuous frame numbers. 
Resolution can be adjusted so that the frame in the every every 5th 30th 100th 
[ every ] or the frame of every multiple frames of other may be displayed, 
therefore, a user - "-- low -- resolving power -- " -- sequential scrolling can be 
carried out using the well-known scroll bar 17, a header, next the highest 
resolving power can be zoomed in in a rough area of interest, and the field can 
be inspected in a detail. 

[0013] Three views of the multimedia information shown in drawing 1 have a 
dialog mutually, and can influence each other. If "caret" 14 used in order to direct 
the inserting point in a storyboard 13 are moved, since the location of this caret is 
reflected, a strip viewer will update oneself. This is shown in drawing 1 and the 
caret 14 of a storyboard 13 is set up between the 1st shot and the 2nd shot. This 
location of a caret 14 is reflected into the sthp viewer 15, and the strip viewer 
shows the boundary of the frame of the 1st shot and the 2nd shot. The innovative 
tool called "SUPURINDA (splinder)" 1 is between the frame of the 1st shot, and 
the frame of the 2nd shot. It can be used in order to adjust a boundary, as 
SUPURINDA 1 is used in order to mark the boundary of a shot, and it is 
explained below. Similarly, the adjustment performed in the field of the strip 
viewer 15 also affects the field of a storyboard 13. For example, if the caret 19 of 



the strip viewer 15 is moved, since a motion of the caret 19 in a strip viewer is 
reflected, the caret 14 of a storyboard 13 will be updated. A player 1 1 can be 
used and the video data chosen with either the storyboard 13 or the strip viewer 
15 can be reproduced. Moreover, in order to specify the segment which should 
be edited by setting up a start point and an ending point while the player 1 1 
shows video, it is also possible to use a player 1 1 . 

[0014] The strip viewer 15 is not limited to a display and edit of video information. 
The interface of the strip viewer 15 is fully common so that any numbers of media 
trucks with which molds differ may be shown. Another example of the strip viewer 
which displays three different trucks 21 of media information, i.e., video, an audio 
23, and three trucks of the graphic form overlay 25 is shown in drawing 3 . It is 
also possible to be also able to edit according to an individual, respectively, and 
to edit three media trucks collectively. As a media truck otherwise considered, 
there are a comment truck, a video special effect truck, a 35mm slide truck, and 
(in order to emulate a motion) an actuator truck (however, not limited to those 
trucks). The media information which indicates by vision at a screen is 
memorized by the related database in fact. The database used in order to 
memorize information is explained further below at a detail. 
[0015] Down the three trucks of the multimedia information on the strip viewer of 
drawing 3 , the scale 27 which displays a frame number or a time basis is located. 
Between each frame 542 and frame 543 of three media trucks (21, 23, and 25), 
the icon called "SUPURINDA" (la, lb, and 1c) is arranged. This location 
expresses the boundary of two different shots so that the contents of the video 
frame of the media truck 21 may show. SUPURINDA (la, lb, and 1c) is used in 
order to divide multimedia information into two different shots, and it can extend 
the media information to the shot of one side or another side. The vocabulary 
"SUPURINDA" was created by combining "a splitter (SPLItter)" and the two 
vocabulary "an extender (ExtenDER)." 

[0016] Detailed drawing of SUPURINDA 601 is shown in drawing 4 . 
SUPURINDA 601 is displayed between the video frames belonging to two 



different shots. The termination frame of the 1st shot in the left-hand side of 
SUPURINDA 601 is expressed by the frame 633, and calls this "the 1st shot" 
hereafter. The first two frames of the 2nd shot in the right-hand side of 
SUPURINDA 601 are expressed by the frame 635, and call this "the 2nd shot" 
hereafter. The icon showing SUPURINDA consists of a head 603, the crus- 
sinistrum-diaphragmatis section 605, and the right trail section 607. Each part of 
the head 603 of SUPURINDA, the crus-sinistrum-diaphragmatis section 605, and 
the right trail section 607 is the "hot spot" which can be caught and pulled apart 
with cursor controllers, such as a mouse, for example, the thing for which a 
mouse is moved until cursor comes to the upper part of SUPURINDA ~ 
SUPURINDA - "- catching ™ " ~ what is necessary is just to push a switch, in 
order to report the prehension next Then, SUPURINDA can be pulled apart by 
moving a mouse, pushing a switch. When a switch is released, since activation of 
edit finishes and the edit is reflected, it turns out that the view was updated. As 
for a strip viewer, it is desirable to carry out focusing of the view again in the 
location where forefront edit was performed. When the head 603 of SUPURINDA 
is caught and pulled apart, whole SUPURINDA 601 moves, and the shot (633 
and 635) of the both sides of SUPURINDA is edited. However, when either the 
crus-sinistrum-diaphragmatis section 605 or the right trail section 607 is caught 
and pulled apart, only the amount of [ the leg and / corresponding ] shank moves, 
and only the shot of the side corresponding to it is edited. 

[0017] Drawing 5 (a) - (c) shows edit of the video track by use of SUPURINDA to 
a detail. Drawing 5 (a) In - (c), the video frame which is in the right-hand side of 
"the 1st shot", a call, and cursor about the video frame in the left-hand side of 
SUPURINDA is called "the 2nd shot." Reference of drawing 5 (a) shows some 
drawings of a strip viewer. In (a), a user places cursor 590 on the head 503 of 
SUPURINDA 512, and "holds" the head 503 of SUPURINDA 512 using cursor 
controllers, such as a mouse. Next, whole SUPURINDA is lengthened to the left 
by the head 503 of SUPURINDA 512 so that it may direct with the arrow head 
used since a motion of cursor is expressed. The beginning of the shot in the 



right-hand side of SUPURINDA 512 is extended at the same time it trims the end 
of the shot which is in the left-hand side of SUPURINDA 512 by lengthening 
SUPURINDA 512 to the left by the head. In drawing 5 (a), since the user is 
lengthening the head 503 of SUPURINDA to a grip and the left with cursor 590, 
trimming of the frame belonging to the 1st shot is carried out, and the frame 
belonging to the 2nd shot is extended. It means that the vocabulary "trimming" 
deletes the frame which is not desired. Moreover, it means that the vocabulary 
"an escape" adds a frame to the boundary of a shot. If length of SUPURINDA is 
stopped and SUPURINDA is stopped in a suitable location, a strip viewer will 
memorize the performed edit. In this example, after a user completes edit, a 
frame is added to one shot, and since it reflects that trimming of the frame of the 
same number was carried out from the shot of another side, a database is 
updated. 

[0018] Drawing 5 (b) shows how SUPURINDA can be used, in order to trim a 
frame only from the end of the 1st shot. In (b), a user lengthens a part for the 
crus-sinistrum-diaphragmatis section 505 and the shank 504 to the left so that 
the crus-sinistrum-diaphragmatis section 505 of SUPURINDA may be directed 
with the arrow head used since cursor 590 expresses a motion of a grip and 
cursor. Trimming is carried out so that the end of the 1st shot may see and show 
a frame by the eye, as dragging of the part for the shank 504 is carried out to the 
crus-sinistrum-diaphragmatis section 505 of SUPURINDA along with the frame of 
the 1st shot, the crus-sinistrum-diaphragmatis section 505 of SUPURINDA ~ the 
— if it is alike shank part 504, and the motion was stopped by the stop and 
SUPURINDA is stopped in a suitable location, a strip viewer will memorize the 
edit. Then, since it reflects that trimming of some frame was carried out from the 
end of the 1st shot, the multimedia database in this example is updated. When 
edit is completed, the "hole" between two shots is lost and SUPURINDA is 
returned to the original configuration. 

[0019] Drawing 5 (c) shows how SUPURINDA can be used, in order to add the 
frame beforehand cut into the 1st shot. In (c), the crus-sinistrum-diaphragmatis 



section 505 of SUPURINDA is moved to a grip, he moves it to the right with 
cursor 590, and the user is extending the end of the 1st shot. It is a wrap 
temporarily about the frame which the frame "was cut" beforehand from the end 
of the 1st shot appears, and belongs to the 2nd shot as a part for the crus- 
sinistrum-diaphragmatis section 505 of SUPURINDA and the shank 504 is 
moved to the right. If the crus-sinistrum-diaphragmatis section 505 of 
SUPURINDA and the motion for the shank [ 504 ] are stopped by the stop and 
SUPURINDA is stopped in a suitable location, the edit is memorized, and since it 
reflects that the frame joined the end of the 1st shot, a database will be updated. 
If the frame of the 1st shot is moved to the left when edit is completed, the frame 
covered temporarily [ the 2nd shot ] will appear and SUPURINDA will return to 
the original configuration. 

[0020] Instead of for a shank [ the left leg ], actuation which does effect in the end 
of the 1st shot shown in drawing 5 (b) and (c) also by the 2nd shot beginning can 
be similarly performed by using a part for the right leg and a shank. For example, 
when the leg on the right of SUPURINDA is caught and SUPURINDA is moved 
to the right along with the frame of the 2nd shot, trimming will be carried out from 
a video frame from the start of the 2nd shot. When similariy a grip is met in the 
right trail section of SUPURINDA, the frame of the 1st shot is met in 
SUPURINDA and it moves to the left, the frame cut previously will be added at 
the beginning of the 2nd shot. 

[0021] When a user "edits" multimedia information on a display screen using 
SUPURINDA of this invention, "edit" on a screen is used and the related 
database which memorizes multimedia information is updated. One example of 
an usable digital video database is shown in drawing 6 . However, the edit-on 
screen technique explained here may use other database structures rather than 
is limited to this specific implementation gestalt of a multimedia database. 
[0022] Reference of drawing 6 shows the implementation gestalt of an usable 
digital video database. There are three rectangles 600,605 and 610 showing a 
compression digital video data file located in a line in the center of drawing. Each 



video-data file contains the compression digital data which specifies the frame of 
video. Tree DS 615 showing the multimedia production which edited is above a 
video-data file. Since a tree structure is very close to the hierarchy whom the 
producer of video or a film is using in order to edit video and a film project as 
shown in drawing 2 , it is desirable. 

[0023] There is DS 620. 625. 630, 635, 640, and 645 which specifies the 
segment of the continuous video frame called a "shot" in the minimum level of 
tree DS 615. Those shot DS contains two pointers. The 1st pointer in each shot 
DS directs the 1st frame of one shot in a video-data file, and the 2nd pointer 
directs the last frame of the shot in the same video-data file. For example, if it 
says about shot DS 620, the 1st pointer 650 will direct the 1st frame 655 of the 
shot contained in the video-data file 600, and the 2nd pointer 660 will direct the 
last frame 665 of the shot. In this example, when performing edit which extends 
or trims a shot, the pointer of the video-data file which specifies beginning or the 
end of that shot is moved appropriately. 

[0024] The DS of high level is above shot DS 620-645 rather than it specifies 
organization of the production after edit. The production consists of two or more 
coordinated DS which forms tree DS. The production root 675 showing the whole 
production is located at the topmost part of a tree. Sequential division is carried 
out at the smaller subunits 680, 685, and 690, and a production results in shot 
DS 620-645 which forms the base of a tree structure. Since a compulsory 
hierarchy is not imposed in this invention, a tree structure presents different 
appearance for every project reflecting how a specific producer composes a 
project. 

[0025] Under the compression digital video files 600-610, there are "virtual 
filmstrip" 700 showing the current sequence of a video data. A shot is 
continuously shown in order so that it may be prescribed by the tree DS of a 
production, as drawing shows. In order to give informational linkage, pointers 705, 
710, 715, 720, 725, and 730 are offered from video files 600, 605, and 610 to the 
virtual filmstrip 700. 



[0026] However, the edit-on screen method of this invention is not linnited to the 
connpression video-data base shown here, for exannple, - instead, probably, it 
will also be possible to change the database which consists of the SMTPE time 
amount code used by edit in order to specify the "Inn point" and the "out point" 
about an actual video tape or a film stock. In the above, multimedia information 
was observed and the approach and equipment to edit were explained. The 
ingredient and array of a component of this invention are considered that it can 
carry out deformation and modification by this contractor, without deviating from 
the meaning of this invention. 
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[Brief Description of the Drawings] 

[Drawing 1] Drawing of the multimedia display edit system containing a player, a 
storyboard, and a strip viewer display. 

[Drawing 2] Drawing showing one of the usable layered structures about a 
multimedia production. 

[Drawing 3] Drawing showing one example of a strip viewer display. 
[Drawing 4] Drawing showing "SUPURINDA" between the video frames 
belonging to two different shots in a detail. 

[Drawing 5] Drawing showing SUPURINDA used in order to edit the shot of video 
frame data. 

[Drawing 6] The video-data base created by this invention and the conceptual 

diagram of DS. 

[Description of Notations] 

1, 1a, 1b, 1c SUPURINDA 

11 Player 

1 3 Storyboard 



14 19 Caret 

15 Strip Viewer 
17 Scroll Bar 

21, 23, 25 Media truck 
27 Scale 

503 Head 

512 SUPURINDA 

504 A Part for Shank 

505 Crus-Sinistrum-Diaphragmatis Section 
590 Cursor 

601 SUPURINDA 
603 Head 

605 Crus-Sinistrum-Diaphragmatis Section 
607 Right Trail Section 
633,635 Frame 



